Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store JSON in database in compact format #53364

Merged
merged 2 commits into from
Jul 22, 2021
Merged

Store JSON in database in compact format #53364

merged 2 commits into from
Jul 22, 2021

Conversation

frenck
Copy link
Member

@frenck frenck commented Jul 22, 2021 •
edited

Proposed change

A great observation from @slovdahl reported on Discord:

image

MariaDB [home_assistant]> SELECT event_type, event_data FROM events WHERE event_type = 'knx_event' LIMIT 1;
+------------+---------------------------------------------------------------------------------------------------------------------------+
| event_type | event_data                                                                                                                |
+------------+---------------------------------------------------------------------------------------------------------------------------+
| knx_event  | {"data": [7, 78], "destination": "1/3/0", "direction": "Incoming", "source": "1.1.34", "telegramtype": "GroupValueWrite"} |
+------------+---------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)

This PR compacts the JSON a bit, thanks @slovdahl ❤️

Result:

image

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

  • No score or internal
  • 🥈 Silver
  • 🥇 Gold
  • 🏆 Platinum

To help with the load of incoming pull requests:

@project-bot project-bot bot added this to Needs review in Dev Jul 22, 2021
@frenck frenck added the smash Indicator this PR is close to finish for merging or closing label Jul 22, 2021
bdraco
bdraco approved these changes Jul 22, 2021
View reviewed changes
Copy link
Member

@bdraco bdraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

Dev automation moved this from Needs review to Reviewer approved Jul 22, 2021
@raman325
Copy link
Contributor

this will only impact new events and states that get recorded right? Is it worth doing a migration as part of this? Perhaps not since the old data should age out over time

@bdraco
Copy link
Member

bdraco commented Jul 22, 2021

this will only impact new events and states that get recorded right? Is it worth doing a migration as part of this? Perhaps not since the old data should age out over time

I think its best not to re-write history if we can avoid it since you are only saving 3-7% and having to pay a large I/O & cpu cost. Probably best to let it expire over time

@bdraco
Copy link
Member

bdraco commented Jul 22, 2021 •
edited

This will break some of the logbook optimizations (we jump through some hoops to avoid de-serializing JSON since we could end up with 200-millions of json deserialize calls per logbook api call) but should be trivial to fix.

diff --git a/homeassistant/components/logbook/__init__.py b/homeassistant/components/logbook/__init__.py
index 0a36790746..e3bb8fcd77 100644
--- a/homeassistant/components/logbook/__init__.py
+++ b/homeassistant/components/logbook/__init__.py
@@ -48,10 +48,10 @@ from homeassistant.helpers.integration_platform import (
 from homeassistant.loader import bind_hass
 import homeassistant.util.dt as dt_util
 
-ENTITY_ID_JSON_TEMPLATE = '"entity_id": "{}"'
-ENTITY_ID_JSON_EXTRACT = re.compile('"entity_id": "([^"]+)"')
-DOMAIN_JSON_EXTRACT = re.compile('"domain": "([^"]+)"')
-ICON_JSON_EXTRACT = re.compile('"icon": "([^"]+)"')
+ENTITY_ID_JSON_TEMPLATE = '"entity_id": ?"{}"'
+ENTITY_ID_JSON_EXTRACT = re.compile('"entity_id": ?"([^"]+)"')
+DOMAIN_JSON_EXTRACT = re.compile('"domain": ?"([^"]+)"')
+ICON_JSON_EXTRACT = re.compile('"icon": ?"([^"]+)"')
 
 ATTR_MESSAGE = "message"

@frenck
Copy link
Member Author

frenck commented Jul 22, 2021

WUT?! haha
omg

@frenck frenck marked this pull request as draft July 22, 2021 21:09
@bdraco
Copy link
Member

bdraco commented Jul 22, 2021 •
edited

WUT?! haha
omg

Yeah..... I'd rather we could normalize more of data in the database instead of JSON, but thats a whole rabbit hole of doom.

@frenck frenck marked this pull request as ready for review July 22, 2021 21:28
@bdraco bdraco merged commit c875ff8 into dev Jul 22, 2021
Dev automation moved this from Reviewer approved to Done Jul 22, 2021
@bdraco bdraco deleted the frenck-2021-2049 branch July 22, 2021 22:06
@github-actions github-actions bot locked and limited conversation to collaborators Jul 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bdraco bdraco bdraco approved these changes

@raman325 raman325 raman325 approved these changes

Assignees
No one assigned
Labels
cla-signed core integration: recorder new-feature small-pr PRs with less than 30 lines. smash Indicator this PR is close to finish for merging or closing
Projects
Dev
  
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@frenck @raman325 @bdraco @homeassistant